VIDEO IMAGE PRODUCING METHOD AND APPARATUS 

BACKGROUND OF THF T NVKNTION 



Field of the Invention 

The present invention relates to a video image producing method and 
apparatus, particularly to a video image producing method and apparatus for 
taking a video by overlapping a subject with a background image. 

Description of Related Art 

Japanese Patent Provisional Publication No. 8-79792 discloses a 
conventional chroma key system. A subject is taken by a video camera with a 
background in a particular color such as blue. A key signal is produced by 
extracting the particular color from the taken image. The image of the subject 
and a background image that is suitably selected among a plurality of prepared 
background images are combined by the chroma key combination with the key 
signal, then the combined image is recorded in a video tape. 

However, the chroma key combination is troublesome in that it causes a 
jitter at the boundary between the subject image and the background image, and 
the combination appears to be artificial because the particular color composition 
for obtaining the key signal overlaps with the background image sometimes. 
Moreover, the subject cannot perceive the video taking like a real experience, 
and gets bored during the taking. 

SUMMARY OF THE INVENTION 

The present invention has been developed in view of the above-described 
disadvantages, and has as its object the provision of the video image producing 



method and apparatus by which the subject image can be naturally combined in 
the background image and at the same time the subject can perceive the video 
taking as if it is a real experience. 

In order to achieve the above-described objects, the present invention is 
directed to a video image producing method, comprising the steps of: displaying 
a movie as a background image on a screen; and taking video by overlapping a 
subject with the background image. Thereby, the subject can be naturally 
combined with the background image, and the subject can perceive the video 
taking like a real experience while looking at the background image during the 
taking. 

Preferably, a treadmill is provided to a floor on which the subject stands; 
and speed of the treadmill and moving speed of the background image are 
controlled in order to be synchronized. Thereby again, the subject can have a 
virtual experience such as strolling at a tourist resort, while obtaining the 
combined image. 

Preferably, the speed of the treadmill and the speed of the background 
image are changed by the subject with a remote control operation. Thus, the 
subject can walk at a desired speed and can stop, so an even more naturally- 
looking combined image can be obtained. 

Preferably, at least one of color temperature and brightness of an 
illumination light illuminating the subject is changed in accordance with the 
background image. In other words, the color temperature and/or brightness of 
the subject can be adjusted with the color temperature and/or brightness of the 
background image, thereby a naturally-looking image combination is possible. 

Preferably, sound is reproduced so as to record the sound with the video 

taking. 

In order to achieve the above-described objects, the present invention is 
directed to a video image producing method, comprising the steps of: displaying 
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an image corresponding with a desired tune on a screen as a background image 
while reproducing auto-accompaniment representing the tune; and taking video 
by overlapping a player with the background image while recording the auto- 
accompaniment and sound of the player. Thereby, the subject can have a virtual 
5 experience such as playing with accompaniment of the orchestra, and the 
experience can be taken in video and its sound can be recorded. 

Preferably, if the tempo of the auto-accompaniment is changed, the 
background image is reproduced corresponding with tempo of the auto- 
i=i accompaniment. 

!SlO In order to achieve the above-described objects, the present invention is 

;JJ directed to a video image producing apparatus, comprising: a video taking box 
!-! which accommodates a subject; a projector which displays an image on a screen 
u being a wall of the video taking box; an image storing device which stores a 
5 sequence of background images; a projector control device which reads out the 
j~15 background images from the image storing device and outputs the background 

3 image to the projector; and a video camera which is provided at a position to face 

u 

at the screen, wherein the video image producing apparatus takes video by 
overlapping the subject with the background image. 

Preferably, the video image producing apparatus further comprises: a 
20 speaker which is provided within the video taking box; an audio storing device 
which stores audio data; an audio reproducing device which reproduces sound by 
reading out the audio data from the audio storing device and outputs audio data 
to the speaker; and a recording device with a microphone which records sound in 
the video taking box. 

25 Preferably, the video image producing apparatus further comprises: a 

treadmill which is provided to a floor of the video taking box; and a control 
device which controls speed of the treadmill and moving speed of the 
background image such that the speeds are synchronized. 
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Preferably, the video image producing apparatus further comprises: a 
remote control which controls at least the speed of the treadmill; a receiver 
which receives a remote control signal from the remote control; and a control 
device which controls the speed of the treadmill in accordance with the remote 
5 control signal received by the receiver, controls the reading out of the 
background image from the image storing device and synchronizes the speed of 
the treadmill and the moving speed of the background image. 

Preferably, the video image producing apparatus further comprises: an 
* illumination device which changes at least one of color temperature and 
(10 brightness of an illumination light illuminating the subject; and an adjusting 
! device which adjusts the at least one of the color temperature and brightness of 
: the illumination device in accordance with the at least one of the color 
■ temperature and brightness of the background image. 

j Preferably, the video image producing apparatus further comprises: a ring 

15 light which encloses the video camera as the illumination device for illuminating 

i 

\ the subject. 

i 

Preferably, the entire wall of the video taking box is the screen, and a 
plurality of pairs of video cameras and illumination devices are built in the 
screen. 

20 Preferably, the video image producing apparatus further comprises: a 

switch device which selectively switches one pair of the plurality of pairs of 
video cameras and illumination devices to be operable. 



BRIEF DESCRIPTION OF THE DRAWINGS 
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The nature of this invention, as well as other objects and advantages 
thereof, will be explained in the following with reference to the accompanying 
drawings, in which like reference characters designate the same or similar parts 
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throughout the figures and wherein: 

Fig. 1 is a plan view showing a schematic appearance of a video image 
producing apparatus of the present invention; 

Fig. 2 is a section view along the line 2-2 in Fig. 1; 
5 Fig. 3 is a front view of an operation panel; 

Fig. 4 is a block diagram showing an embodiment of the video image 
producing apparatus of the present invention; 

Fig. 5 is a plan view of a remote control; 
q Figs. 6(A) and 6(B) are a side view and a front view of a color 

jSlO temperature conversion part; 

j^j Fig. 7 is a flowchart showing the overall process of producing a video 

image of the present invention; 
- Fig. 8 is a flowchart used for depicting controls of respective parts 

^ associated with remote control; 

•^15 Fig. 9 is a flowchart used for depicting controls of respective parts 

[ 3 associated with remote control; 

Fig. 10 is a flowchart used for depicting controls of respective parts 
associated with remote control; 

Fig. 11 is a flowchart used for depicting the control over illumination 
20 intensity and color temperature; 

Fig. 12 is a plan view showing a schematic appearance of another 
embodiment of the video image producing apparatus of the present invention; 
Fig. 13 is a section view along the line 13-13 in Fig. 12; 
Fig. 14 is a block diagram showing an embodiment of a video image 
25 producing apparatus of the present invention; and 

Fig. 15 is a flowchart showing a process for video taking by overlapping 
a subject (player) with a background image while recording auto-accompaniment 
and music of the player. 
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Hereunder a preferred embodiment of a video image producing method 
and apparatus will be described with reference to the accompanying drawings. 

Fig. 1 is a plan view showing a schematic appearance of the video image 
producing apparatus according to an embodiment of the present invention. Fig. 
2 is a section view along the line 2-2 in Fig. 1. 

As shown in Figs. 1 and 2, the video image producing apparatus has a 
video taking box 12, which is provided in a studio 10 with an entrance 10A, and 
screens S1-S4 are provided on the four walls of the video taking box 12. Video 
cameras Cam 1-Cam 4 with illumination lights L1-L4 are built in the screens Sl- 
S4, respectively. The illumination lights L1-L4 are preferable to be ring lights 
in order to avoid the subject's shadow from being taken. Projectors P1-P4 are 
disposed for projecting a background image on the screens S1-S4 to a subject, 
respectively. The video cameras Cam 1-Cam 4 are provided with microphones 
M1-M4 (see Fig. 4) and remote-control light receivers R1-R4 (see Fig. 4), 
respectively. 

A treadmill 14 is disposed on the floor of the video taking box 12, and 
the subject 16 can walk or run in place on a continuous moving belt of the 
treadmill 14. The treadmill 14 is driven in the direction of an arrow A or an 
arrow B with operation of a remote control 18 by the subject 16 as shown in Fig. 
2, and the moving speed of the treadmill 14 can be changed. Detailed 
description on the control of the treadmill 14 by the remote control and other 
controls will be described later. 

An operation panel 20 is provided at the front of the studio 10. As 
shown in Fig. 3, the operation panel 20 is provided with a CRT display 21, an 
opening 22 for inserting and ejecting a record medium (e.g., a videocassette) for 
recording the video image, a print ejecting opening 23, an opening 24 for 
receiving paper money and coins, a tray 25 for giving change, and an operation 



part 26 including types of buttons, and so forth. 

Fig. 4 is a block diagram showing an embodiment of the video image 
producing apparatus according to an embodiment of the present invention. 

The video image producing apparatus comprises the video cameras Cam 
1-Cam 4 with the illumination lights L1-L4, the projectors P1-P4, the 
microphones M1-M4 and the remote-control light receivers R1-R4, which are 
provided to the cameras Cam 1-Cam 4, respectively, the treadmill 14, color 
temperature converting parts T1-T4, an audio reproducing circuit 30, a recording 
and reproducing device 46, a central processing unit (CPU) 50, a light intensity 
controller 52, an illumination color temperature controller 54, a background 
image and audio library 56, a background image generator 62, a projector 
controller 64, and so forth. 

Fig. 5 is a plan view of the remote control 18 for controlling the treadmill 
14 and so forth. 

The remote control 18 has an up-switch 18A and a down-switch 18B for 
bringing up/down the speed of the treadmill 14 and so forth, a right turn switch 
18C, a left turn switch 18D, a stop switch 18E, manual switches 18F-18I for 
manually selecting the video cameras Cam 1-Cam 4, an automatic switch 18 J for 
automatically switching the video cameras Cam 1-Cam 4, a remote-control light 
emitting part 18K for outputting infrared light as a carrier wave to transmit a 
remote-control signal in response to the switch operation. The remote-control 
light outputted from the remote-control light emitting part 18K is received by 
one or some of the remote-control light receivers R1-R4, which are provided to 
the video cameras Cam 1-Cam 4, respectively. 

The color temperature conversion parts T1-T4 convert color temperatures 
of the illumination light that illuminates the subject. As shown in Fig. 6 (A), 
each of the color temperature conversion parts T1-T4 comprises a light source 70, 
a glass filter 72 arranged in front of the light source 70, and a motor 74 to turn 
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the glass filter 74 at a suitable position. The glass filter 72 is colored in 
gradation from red to blue in a circumferential direction as shown in Fig. 6(B). 

Returning to Fig. 4, the CPU 50 supervises the entire apparatus. The 
CPU 50 receives data indicating an amount of deposited money from a charging 
machine 34. The CPU 50 receives, from each of the remote-control light 
receiving parts R1-R4, an intensity signal indicating the intensity of the remote- 
control light received by the each of the remote-control light receiving parts Rl- 
R4, and also receives, through an A/D converter 48, the remote-control signal 
transmitted by the remote-control light. The CPU 50 receives an instruction 
signal such as a select signal of the background image and a print request signal 
from the operating part 26. 

Operations of the CPU 50 and the parts shown in Figs. 1-4 will be 
described referring to the flowcharts in Figs. 7-11. 

Fig. 7 is a flowchart showing an entire process from receiving money 
from the subject to giving the subject a videocassette in which a video image 
including the subject is recorded. 

The CPU 50 determines whether or not the money is put by the data from 
the charging machine 34 (step S10). If the CPU 50 determines that the money 
is put, the CPU 50 then determines whether or not a videocassette is inserted in 
the insertion opening 22 (step S12). 

As the CPU 50 determines that the videocassette is inserted, the subject 
selects a background image by operating with the buttons at the operating part 26 
(step S14). When the background image is selected, video taking is started 
(step S16). In other words, when the background image is selected, the CPU 50 
reads out a corresponding background image signal and an associated audio 
signal from the background image and audio library 56. Then, the CPU 50 
outputs the read background image signal to the background image generator 62, 
and outputs the read audio signal to the audio reproducing circuit 30. The 
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background image signal and the audio signal can be taken into the background 
image and audio library 56 from an external server 58 via a communication 
device 60. 

The background image generator 62 outputs the background image signal, 
5 which is inputted from the background image and audio library 56, to the 
projectors P1-P4 through the projector controller 64 in accordance with the 
control signal from the CPU 50. The projectors P1-P4 project the background 
image on the four screens S1-S4 in accordance with the inputted background 
3 image signal, respectively. 

; SlO The audio reproducing circuit 30 converts the audio signal, which is 

inputted from the background image and audio library 56, from a digital form to 
an analog form, and amplifies and outputs the analog audio signal to a speaker S; 

u thus the sound is reproduced. 

k % On the other hand, each of the video cameras Cam 1-Cam 4 takes a video 

|^15 image by overlapping the subject with the background image that is projected on 
[ 3 the screen that is facing to the correspond camera, and outputs the video signal to 
a camera input selector 44. The camera input selector 44 selects one of the four 
video signals, which are inputted from the video cameras Cam 1-Cam 4, in 
accordance with the selecting instruction from the CPU 50, and outputs the 
20 selected video signal to the recording and reproducing device 46. 

Each of the microphones M1-M4 picks up sound of the video taking box 
12, and outputs the audio signal to a microphone input selector 38. The 
microphone input selector 38 selects one of the four audio signals, which are 
inputted from the microphones M1-M4, in accordance with the selecting 
25 instruction from the CPU 50, and outputs the selected audio signal to an audio 
combining circuit 40. The audio combining circuit 40 combines the audio 
signal that is inputted from the microphone input selector 38 and the audio signal 
such as background music, which is outputted from the background image and 
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audio library 56, and outputs the combined audio signal to the recording and 
reproducing device 46. The recording and reproducing device 46 records the 
video signal and the audio signal in the videocassette. 

When the video taking in a predetermined time length is completed (step 
5 S18) as described above, the CPU 50 determines whether or not printing is 
required (step S20). If printing is required, the CPU 50 selects an image to be 
printed (step S22) as follows: the video signal is reproduced from the recording 
and reproducing device 46 so as to display a combined image of the subject and 

: ^ the background image on an image display 36, and the user operates forward 

;^10 feeding of frames and stopping a frame. When the image to be printed is 
selected, the recording and reproducing device 46 outputs the video signal 

; r : representing the frame image to the printing device 42 so as to let the printing 

u device 42 print the still image (step S24). 

3 If no printing is required or printing is completed, the CPU 50 determines 

j=fl5 whether or not charge settlement is required (step S26). If no settlement is 
□ required, the process returns to step S20; if the settlement is required, the 
videocassette is ejected through the ejection opening 22, and the CPU 50 settles 
the charge and returns the change (step S30). 

Next, controls of the respective parts associated with the remote control 
20 operation will be described referring to flowcharts in Figs. 8-10. 
i As shown in Fig. 2, the subject 16 carries the remote control 18 and 

stands on the treadmill 14. The subject 16 operates the remote control 18 when 
necessary, and the CPU 50 determines whether or not any switch or button of the 
remote control 18 is pressed (step S40 in Fig. 8). If no switches or buttons of 
25 the remote control 18 are pressed, the CPU 50 determines whether or not which 
video camera is manually selected among the video cameras Cam 1-Cam 4 with 
the microphones M1-M4 (in short, whether or not the video camera to be used 
for video taking is set) (step S42). 
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If the video camera is set, which means, if one of the manual switches 
18F-18I (see Fig. 5) for manually selecting one of the video cameras Cam 1-Cam 
4 is turned on, the CPU 50 returns to step S40 and controls the camera input 
selector 44 and the microphone input selector 38 in such a manner as to taking 
the video signal and the audio signal from the video camera and the microphone 
that are selected by the manual switches 18F-18I. 

If the video camera is not set, which means, if the automatic switch 18 J 
for automatically switching the video cameras Cam 1-Cam 4 is turned on, the 
CPU 50 obtains the signal indicating the intensity of the received remote-control 
light in accordance with the remote control signal inputted from the remote- 
control light receiving parts R1-R4 of the video cameras (step S44). The CPU 
50 then switches the camera input selector 44 and the microphone input selector 
38 so that the video camera and the microphone with the highest intensity of 
received remote-control light is selected (step S46). 

If the CPU 50 determines that the switches or the buttons of the remote 
control 18 are pressed at step S40, the process goes on to a subroutine in Fig. 9 
(step S48). 

As shown in Fig. 9, the CPU 50 determines whether or not the up-switch 
18 A or the down-switch 18B of the remote control 18 for bringing up or down 
the speed of the treadmill 14 is pressed (step S50). If the CPU 50 determines 
that the up-switch 18A or the down-switch 18B is pressed, the CPU 50 outputs 
the control signal to the drive circuit 32 so as to bring up or down the speed of 
the treadmill 14, which moves in an instructed direction, and changes the speed 
of the treadmill 14 (step S52). 

The CPU 50 brings up or down the speed of the treadmill 14 in a step-by- 
step manner by pressing the up-switch 18A or the down-switch 18B once. The 
moving direction of the treadmill 14 is determined in accordance with an thrown 
direction of the remote control light (the direction in which the subject is 
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moving). In other words, as shown in Fig. 10, the intensity is obtained of the 
received remote-control light, which enters into the remote-control light 
receiving parts R1-R4 of the video cameras Cam 1-Cam 4 (step S62), and the 
CPU 50 determines whether or not the video camera with the highest intensity of 
the received remote-control light is the video camera Cam 1 or Cam 2 (step S64). 

As shown in Fig. 1, if the video camera with the highest intensity is the 
video camera Cam 1 or Cam 2, it means that the subject is moving in the 
direction of the arrow B, and the treadmill 14 is then driven in the direction of 
the arrow A (step S66), so that the subject can stand at substantially the center of 
the treadmill 14 all the time (see Fig. 2). In a case where the video camera with 
the highest intensity is not the video camera Cam 1 or Cam 2 (i.e., the video 
camera with the highest intensity is the video camera Cam 3 or Cam 4), it means 
that the subject is moving in the direction of the arrow A; hence the treadmill 14 
is driven to move in the direction of the arrow B (step S68). 

At step S52 in Fig. 9, the CPU 50 controls the treadmill 14, and at the 
same time controls the moving direction, speed, and sound of the background 
image correspondingly with the moving direction and moving speed of the 
treadmill 14. 

The CPU 50 determines at step S54 whether or not the right-turn switch 
18C or the left-turn switch 18D of the remote control 18 is pressed. If the CPU 
50 determines that the right-turn switch 18C or the left-turn switch 18D is 
pressed, it changes the background image and sound in the instructed direction 
(step S56). The subject is taken by the video as if he/she turned right or left by 
the changing of the direction of the background image. 

The CPU 50 determines at step S58 whether or not the stop switch 18E of 
the remote control 18 is pressed. If the CPU 50 determines that the stop switch 
18E is pressed, it stops the treadmill 14 (step S60). 

Now, controls over the intensity of illumination and color temperature 
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will be described referring to a flowchart in Fig. 11. 

First, the CPU 50 obtains color temperature data of the selected 
background image, and outputs the obtained color temperature data to the 
illumination color temperature controller 54 (step S70). The illumination color 
temperature controller 54 determines a rotation position of the glass filter 72 (see 
Figs. 6(A) and 6(B) ) according to the obtained color temperature data (step S72), 
and controls the motor 74 so that the glass filter 72 moves at the determined 
position (step S74). Thereby, the color temperature of the illumination light, 
which illuminates the subject, is adjusted to correspond with the color 
temperature of the background image. 

Moreover, the CPU 50 also obtains average brightness data of the 
selected background image, and outputs the obtained average brightness data to 
the illumination intensity controller 52 (step S76). The illumination intensity 
controller 52 determines the brightness of the illumination light by referring to 
the received average brightness data (step S78), and adjusts the brightness of the 
illumination light to be the determined brightness (step S80). 

Fig. 12 is a plan view of a schematic appearance of another embodiment 
of the video image producing apparatus of the present invention. Fig. 13 is a 
section view along the line 13-13 in Fig. 12. 

As shown in Figs. 12 and 13, a plurality of small-sized speakers S are 
built in the walls of the studio 80 of the video image producing apparatus in a 
form of an array, and one projector P, which uses one wall as a screen, and one 
video camera Cam with two ring lights and two illumination lights L are 
arranged in the studio 80. In this embodiment, the piano 82 is set in the studio 
80. 

Fig. 14 is a block diagram showing an embodiment of the video image 
producing apparatus of the present invention. 

The video image producing apparatus comprises the video camera Cam 
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with the ring light, the projector P, the microphone M provided to the video 
camera Cam, the illumination light L, the audio reproducing circuit 30, a 
personal computer 90 including a keyboard, which serves as a play condition 
input device, and a monitor, a charging machine 92, a CPU 94, a database 96, 
which records the background image signal indicating an image of the play and 
the audio signal as the auto-accompaniment of every tune, an image processing 
device 98, an audio reproducing circuit 100, and a play recording device 102. 

Fig. 15 is a flowchart showing a process for taking video by overlapping 
the subject (player) with the background image and for recording the auto- 
accompaniment and the playing sounds of the player. 

When the player puts a predetermined amount of money in a charging 
machine 92, a process shown in Fig. 15 is started (step S100), and the player first 
selects an instrument and a tune to play (steps S102 and S104). The player then 
selects the playing condition such as pitch and tempo of the selected tune (step 
S106). 

Next, the player selects a scene as the background image (step S108). 
More specifically, there are views from audience sheets to the stage and from the 
stage to the sheets and so forth, and the player selects a desired scene among the 
scenes. The selecting operations through steps S102-S108 are performed using 
the keyboard and the mouse of the personal computer 90. 

The CPU 94 reads out the background image signal of the selected scene 
from the database 96 and outputs the selected background image signal to the 
image processing device 98. The image processing device 98 converts the 
background image signal, which is taken from the database 96, to be congruous 
with the playing condition (tempo), and outputs the background image signal to 
the projector P (step S110). This process is necessary in purpose of adjusting 
the movement of the background image with the tempo of the play. 

The CPU 94 also reads out from the database 96 the audio signal of the 
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respective instruments that play the selected tune (this reading, however, 
excludes the audio signal of the instrument that is actually played by the player), 
and lets the audio reproducing circuit 100 output the audio signal. The audio 
reproducing circuit 100 converts the audio signals of the respective instruments 
that are taken from the database 96 from the digital form to the analog form and 
amplifies the audio signals, then outputs the audio signals to the corresponding 
speaker S among the plural speakers S, which are arranged like an array so as to 
reproduce the auto-accompaniment. Thereby, for example, one speaker S 
generates the sound of the violin while another speaker generates the sound of 
the clarinet; and thus the player can perceive the virtual experience like a real 
one as if he/she is standing on the stage with an orchestra. 

As described above, the player can play one's instrument with the auto- 
accompaniment. When the play is started, the video camera Cam takes a video 
image by overlapping the player with the background image during the play and 
outputs the video signal to the play recording device 102, and at the same time 
the microphone M picks up the auto-accompaniment and the playing sound and 
outputs the audio signal to the play recording device 102 (step SI 02). Thereby, 
an image in which the player is overlapped with the background image is 
recorded while the auto-accompaniment and the playing sounds of the player are 
recorded, and when the play is finished, the recording of the image and sound is 
completed (step S104). 

As described above, according to the present invention, a moving image 
as a background image is displayed on the screen and the subject is videotaped 
by overlapping the subject with the background image. Thus the subject can 
naturally be combined in the background image, and further, the subject can 
stroll in a tourist resort, a computer graphic, and a virtual reality space during 
video taking, or can enjoy a virtual experience such as playing one's instrument 
with an orchestra. 
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It should be understood, however, that there is no intention to limit the 
invention to the specific forms disclosed, but on the contrary, the invention is to 
cover all modifications, alternate constructions and equivalents falling within the 
spirit and scope of the invention as expressed in the appended claims. 



